feat: reset all Jobs to a single pinned 2.0.0 release - #35
Merged
Conversation
Move the whole repository off the mutable 'stable'/'latest' tags and onto a single immutable release tag, so a Job that ran in July still runs, and gives the same results, in December. - All 48 Job Definitions across the 8 files in data-manager/ go to version '2.0.0' (they were a mix of 1.0.0, 1.0.1 and 1.0.2), and the 46 image tags for our own images go to '2.0.0'. - The 28 Nextflow process 'container' directives in nf-processes/ are pinned to :2.0.0 to match. These are COPY'd into the vs-nextflow image, so a :2.0.0 vs-nextflow now spawns :2.0.0 process containers. - Dockerfile-fns, -moldb, -mordred, -oddt and -prep are pinned to informaticsmatters/vs-rdkit-base:2.0.0, so the release images do not sit on a moving base. That base image is built by hand and must be published before the release workflow runs. - New manually-dispatched publish-tag workflow builds and pushes all 11 images under a given tag, calling the existing build-all-with-push-option.yaml. build.yaml, publish-latest.yaml and publish-stable.yaml are unchanged - the latest/stable development loop still has value for the non-Nextflow images. - test.yaml now also runs jote against manifest-moldb.yaml. It was the one manifest CI never validated, and its 10 Jobs are the second largest block of changes here. 2.0.0 rather than 1.0.0: vs-prep:1.0.0, vs-nextflow:1.0.0 and vs-rdock:1.0.0 already exist on Docker Hub (pushed 2021-06) and git tags 1.0.0-1.0.11 are taken. Reusing a static tag would leave Kubernetes nodes that cached the 2021 image silently running the old content. The two silicos-it Jobs move to version 2.0.0 but keep 3dechem/silicos-it:latest - it is a third-party image whose only published tag is 'latest' (2017). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #34.
Moves the whole repository off the mutable
stable/latesttags and onto a single immutable release tag, so the Jobs can be wiped from the Data Manager and re-loaded as one coherent, reproducible set.What changed
data-manager/(8 files)versionvalues →'2.0.0'(were 38 ×1.0.0, 8 ×1.0.1, 2 ×1.0.2); the 46image.tagvalues for our images →'2.0.0'nf-processes/(22 files)container 'informaticsmatters/…'directives →:2.0.0FROM informaticsmatters/vs-rdkit-base:latest→:2.0.0(-fns,-moldb,-mordred,-oddt,-prep).github/workflows/publish-tag.yamlimage-taginput, calls the existingbuild-all-with-push-option.yamlwithimage-push: true.github/workflows/test.yamljote --manifest manifest-moldb.yaml --dry-runmoldb/k8s-create-tables.yaml,README.md,USER_GUIDE.md,docker-compose.yamlWhy
2.0.0and not1.0.0informaticsmatters/vs-prep:1.0.0,vs-nextflow:1.0.0andvs-rdock:1.0.0already exist on Docker Hub (pushed 2021-06), and git tags1.0.0–1.0.11are already taken here. The DM treats any tag other thanlatest/stableas static and caches it per Kubernetes node, so republishing1.0.0would leave some nodes silently running 2021 code.2.0.0is unused across all 11 images and the base — verified against the Docker Hub tags API.Deliberately unchanged
3dechem/silicos-it:latest— third-party, andlatestis the only tag it has ever published (2017-08-15). The two silicos-it Jobs move to version2.0.0but keep that image tag, and stay non-reproducible. The existing explanatory comment is untouched.build.yaml,publish-latest.yaml,publish-stable.yaml— thelatest/stabledevelopment loop still has value for the non-Nextflow images.euclia/jaqpotpy-inference:1.1.0,informaticsmatters/rdock:2013.1,python:3.10.12-slim-bullseye,debian:bullseye— already pinned.kind-version: '2021.1'— schema version, not a Job version.Known consequence
nf-processes/isCOPY'd into thevs-nextflowimage, so these tags are baked in at build time. Thelatest/stablevs-nextflowimages built fromstaging/mainwill now also spawn:2.0.0process containers, which makes the dynamic-tag development loop incoherent for Nextflow Jobs. Accepted for now; the clean fix (a Nextflowparamsvalue) is a follow-up noted in #34, complicated bynextflow.confignot being shipped in the image.Release sequence after merge
informaticsmatters/vs-rdkit-base:2.0.0by hand — five images areFROMit and the CI build fails without it.publish-tagfrommainwithimage-tag: 2.0.0; confirm all 11 images appear on Docker Hub.2.0.0. Never move it.https://raw.githubusercontent.com/InformaticsMatters/virtual-screening/2.0.0/data-manager/manifest-im-virtual-screening.yaml.Verification done
jote --dry-runpasses against all six manifests: fragnet-search 3/3, im-virtual-screening 32/32, silicos-it 0/0, dmpk 1/1, im-mordred 2/2, moldb 12/12. (moldb passes without--allow-no-tests, so it is added to CI unqualified.)tag: 'stable',version: '1.0.,:stableundernf-processes/, andvs-rdkit-base:latestall return nothing. Counts confirmed at 48 / 46 / 2 / 28.2.0.0confirmed absent from Docker Hub for every image, including the base.# tag: …line in the guide comment block atdata-manager/im-virtual-screening.yaml:40was not touched by the sweep.Not yet done: no
2.0.0image exists anywhere yet, so nothing has been run against a real2.0.0container. Step 1 and 3 above have to happen before an end-to-endjoterun (particularly a Nextflow Job such asrun-rdockormoldb-load-library, which is what proves the baked-in process container tags resolve).🤖 Generated with Claude Code